Image segmentation method and apparatus and storage medium

ABSTRACT

An image segmentation method includes obtaining target domain images and source domain images, and segmenting the source domain images and the target domain images by using a generative network in a first generative adversarial network. The method further includes segmenting the source domain images and the target domain images by using a generative network in a second generative adversarial network, and determining a first source domain image and a second source domain image according to source domain segmentation losses, and determining a first target domain image and a second target domain image according to target domain segmentation losses. The method also includes performing cross training on the first generative adversarial network and the second generative adversarial network to obtain a trained first generative adversarial network; and segmenting a to-be-segmented image based on the generative network in the trained first generative adversarial network.

RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/CN2020/124673, filed on Oct. 29, 2020, which claims priority toChinese Patent Application No. 202010084625.6, entitled “IMAGESEGMENTATION METHOD AND APPARATUS AND STORAGE MEDIUM” and filed on Feb.10, 2020. The disclosures of the prior applications are herebyincorporated by reference in their entirety.

FIELD OF THE TECHNOLOGY

This application relates to the field of communication technologies,including an image segmentation method and apparatus and a storagemedium.

BACKGROUND OF THE DISCLOSURE

With the development of artificial intelligence (AI), the application ofAI in the medical field has become more and more extensive. Significantresults have been achieved in various medical image analysis tasks suchas image classification, lesion detection, target segmentation, andmedical image analysis, especially in the segmentation of medicalimages. For example, AI technologies may be applied to segment an opticcup and an optic disc from a retinal fundus image. A current AI solutionfor segmenting an optic cup and an optic disc is mainly based on a deeplearning network. Specifically, a deep learning network that can segmentan optic cup and an optic disc may be trained, then a fundus image to besegmented is inputted into the trained deep learning network to extractfeatures, and an optic cup and an optic disc are segmented based on thefeatures to obtain a segmentation result, for example, a glaucomasegmentation image, or the like.

During the research and practice of the related art, it has been foundthat the performance of a trained deep convolutional neural networkmodel usually degrades during a test on unprecedented data, especiallywhen there is a significant domain shift between training (sourcedomain) data and testing (target domain) data. A domain shift is acommon problem in the biomedical field. Because biomedical images areacquired by different imaging modalities or different settings of thesame device, different acquired images have variability in texture,color, shape, and the like. Therefore, the accuracy of segmentation isnot high.

SUMMARY

According to various embodiments provided in this application, an imagesegmentation method and apparatus and a storage medium are provided.

In an embodiment, an image segmentation method includes obtaining pluraltarget domain images and plural source domain images that are labeledwith target information, and segmenting one or more of the source domainimages and the target domain images by using a generative network in afirst generative adversarial network to respectively determine firstsource domain segmentation losses and first target domain segmentationlosses. The method further includes segmenting one or more of the sourcedomain images and the target domain images by using a generative networkin a second generative adversarial network to respectively determinesecond source domain segmentation losses and second target domainsegmentation losses, and determining a first source domain image and asecond source domain image according to the first source domainsegmentation losses and the second source domain segmentation losses,and determining a first target domain image and a second target domainimage according to the first target domain segmentation losses and thesecond target domain segmentation losses. The method also includesperforming cross training on the first generative adversarial networkand the second generative adversarial network by using the first sourcedomain image, the first target domain image, the second source domainimage, and the second target domain image to obtain a trained firstgenerative adversarial network, and segmenting a to-be-segmented imagebased on the generative network in the trained first generativeadversarial network to obtain a segmentation result.

In an embodiment, an image segmentation apparatus includes processingcircuitry configured to obtain plural target domain images and pluralsource domain images that are labeled with target information, andsegment one or more of the source domain images and the target domainimages by using a generative network in a first generative adversarialnetwork to respectively determine first source domain segmentationlosses and first target domain segmentation losses. The processingcircuitry is further configured to segment one or more of the sourcedomain images and the target domain images by using a generative networkin a second generative adversarial network to respectively determinesecond source domain segmentation losses and second target domainsegmentation losses, and determine a first source domain image and asecond source domain image according to the first source domainsegmentation losses and the second source domain segmentation losses,and determine a first target domain image and a second target domainimage according to the first target domain segmentation losses and thesecond target domain segmentation losses. The processing circuitry isalso configured to perform cross training on the first generativeadversarial network and the second generative adversarial network byusing the first source domain image, the first target domain image, thesecond source domain image, and the second target domain image to obtaina trained first generative adversarial network, and segment ato-be-segmented image based on the generative network in the trainedfirst generative adversarial network to obtain a segmentation result.

In an embodiment, a non-transitory computer-readable storage mediumstores computer-readable instructions thereon which, when executed by aprocessor, cause the processor to perform an image segmentation method.The method includes obtaining plural target domain images and pluralsource domain images that are labeled with target information, andsegmenting one or more of the source domain images and the target domainimages by using a generative network in a first generative adversarialnetwork to respectively determine first source domain segmentationlosses and first target domain segmentation losses. The method furtherincludes segmenting one or more of the source domain images and thetarget domain images by using a generative network in a secondgenerative adversarial network to respectively determine second sourcedomain segmentation losses and second target domain segmentation losses,and determining a first source domain image and a second source domainimage according to the first source domain segmentation losses and thesecond source domain segmentation losses, and determining a first targetdomain image and a second target domain image according to the firsttarget domain segmentation losses and the second target domainsegmentation losses. The method also includes performing cross trainingon the first generative adversarial network and the second generativeadversarial network by using the first source domain image, the firsttarget domain image, the second source domain image, and the secondtarget domain image to obtain a trained first generative adversarialnetwork, and segmenting a to-be-segmented image based on the generativenetwork in the trained first generative adversarial network to obtain asegmentation result.

Details of one or more embodiments of this application are provided inthe accompanying drawings and descriptions below. Other features,objectives, and advantages of this application become apparent from thespecification, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

To describe the technical solutions in the embodiments of thisapplication more clearly, the following briefly describes accompanyingdrawings describing the embodiments. The accompanying drawings in thefollowing descriptions show merely some embodiments of this application,and a person skilled in the art may still derive other accompanyingdrawings from these accompanying drawings.

FIG. 1a is a schematic diagram of a scenario of an image segmentationmethod according to an embodiment of this application.

FIG. 1b is a flowchart of an image segmentation method according to anembodiment of this application.

FIG. 2a is another flowchart of an image segmentation method accordingto an embodiment of this application.

FIG. 2b is a system architecture diagram of an image segmentation methodaccording to an embodiment of this application.

FIG. 2c is an architecture diagram of a first generative adversarialnetwork according to an embodiment of this application.

FIG. 2d is a diagram of an image segmentation result according to anembodiment of this application.

FIG. 3 is a schematic structural diagram of an image segmentationapparatus according to an embodiment of this application.

FIG. 4 is a schematic structural diagram of an electronic deviceaccording to an embodiment of this application.

DESCRIPTION OF EMBODIMENTS

The technical solutions in embodiments of this application are describedin the following with reference to the accompanying drawings in theembodiments of this application. The described embodiments are merelysome, rather than all, of the embodiments of this application. All otherembodiments obtained by a person skilled in the art based on theembodiments of this application shall fall within the protection scopeof this application.

Embodiments of this application provide an image segmentation method andapparatus and a storage medium. The image segmentation method may beintegrated in an electronic device. The electronic device may be aserver or may be a device such as a terminal.

The image segmentation method provided in this embodiment of thisapplication relates to the direction of CV in the Al field, and mayimplement segmentation of a fundus image of the CV technologies of AI toobtain a segmentation result.

AI is a theory, method, technology, and application system that uses adigital computer or a machine controlled by the digital computer tosimulate, extend, and expand human intelligence, perceive anenvironment, acquire knowledge, and use knowledge to obtain an optimalresult. In other words, Al is a comprehensive technology in computerscience and attempts to understand the essence of intelligence andproduce a new intelligent machine that can react in a manner similar tohuman intelligence. Al is to study the design principles andimplementation methods of various intelligent machines, to enable themachines to have the functions of perception, reasoning, anddecision-making. The AI technology is a comprehensive discipline, andrelates to a wide range of fields including both hardware-leveltechnologies and software-level technologies. AI software technologiesmainly include directions such as a computer vision (CV) technology, andmachine learning (ML)/deep learning.

CV technology is a science that studies how to use a machine to “see”,and furthermore, refers to using a computer to replace human eyes forperforming machine vision, such as recognition and measurement, on atarget, and further perform image processing, so that the computerprocesses an image into an image more suitable for human eyes toobserve, or an image transmitted to an instrument for detection. As ascientific discipline, CV studies related theories and technologies andattempts to establish an Al system that can obtain information fromimages or multidimensional data. The CV technology usually includestechnologies such as image processing and image recognition, and furtherinclude biological feature recognition technologies such as common facerecognition and human pose recognition.

In this embodiment of this application, image segmentation is a CVtechnology and process of segmenting an image into several particularareas having special properties, and specifying a target of interest. Inthis embodiment of this application, a medical image such as a fundusimage is mainly segmented, to find a target object. For example, anoptic cup, an optic disc or the like is segmented from a fundus image.The segmented target object may be subsequently analyzed by health careprofessionals or other medical experts to perform a furthercorresponding operation.

For example, referring to FIG. 1a , first, the electronic deviceintegrated with the image segmentation apparatus first obtains a targetdomain image and a source domain image that is labeled with targetinformation (or, for example, plural target domain images and pluralsource domain images), then segments the source domain image and thetarget domain image (or, for example, segments each of the plural targetdomain images and the plural source domain images) by using a generativenetwork in a first generative adversarial network to respectivelydetermine a first source domain segmentation loss (or, for example,first source domain segmentation losses) and a first target domainsegmentation loss (or, for example, first target domain segmentationlosses), segments the source domain image and the target domain image(or, for example, segments each of the plural target domain images andthe plural source domain images) by using a generative network in asecond generative adversarial network to respectively determine a secondsource domain segmentation loss (or, for example, second source domainsegmentation losses) and a second target domain segmentation loss (or,for example, first target domain segmentation losses), next, determinesa first source domain target image (or first source domain image) and asecond source domain target image (or second source domain image)according to the first source domain segmentation loss and the secondsource domain segmentation loss (or, for example, the first sourcedomain segmentation losses and the second source domain segmentationlosses), determines a first target domain target image (or a firsttarget domain image) and a second target domain target image (or asecond target domain image) according to the first target domainsegmentation loss and the second target domain segmentation loss (or,for example, the first target domain segmentation losses and the secondtarget domain segmentation losses), then performs cross training on thefirst generative adversarial network and the second generativeadversarial network by using the first source domain target image, thefirst target domain target image, the second source domain target image,and the second target domain target image to obtain the first generativeadversarial network after training (or a trained first generativeadversarial network), and then segments a to-be-segmented image based onthe generative network in the first generative adversarial network aftertraining to obtain a segmentation result.

In the solution, two generative adversarial networks have differentstructures and learning capabilities and may perform mutual learning andmutual supervision, and a clean target image is selected from onenetwork and provided to the peer network to continue with training,thereby effectively improving the accuracy of image segmentation.

In this embodiment, a description is provided from the perspective ofthe image segmentation apparatus. The image segmentation apparatus maybe specifically integrated in an electronic device. The electronicdevice may be a server or may be a terminal or may be a system includinga server and a terminal. When the electronic device is a systemincluding a server and a terminal, the image segmentation method in theembodiments of this application implements interaction through theterminal and the server.

The terminal may be specifically a desktop terminal or a mobileterminal. The mobile terminal may be specifically at least one of amobile phone, a tablet computer, a notebook computer or the like. Theserver may be implemented by an independent server or implemented by aserver cluster including a plurality of servers.

As shown in FIG. 1b , a specific procedure of the image segmentationmethod may be as follows.

In step 101, a target domain image and a source domain image that islabeled with target information are obtained. In an embodiment, step 101includes obtaining plural target domain images and plural source domainimages that are labeled with target information.

The source domain image is a medical image that may provide richannotation information. The target domain image is a medical image thatbelongs to the field of a test dataset and lacks annotation information.For example, the source domain image may be specifically obtained byperforming image acquisition on the tissue of a living body by a medicalimage acquisition device such as a Computed Tomography (CT) device or aMagnetic Resonance Imaging (MRI) device. The image is annotated by aprofessional, for example, annotated by an imaging physician andprovided to the image segmentation apparatus. That is, the imagesegmentation apparatus may specifically receive a medical image sampletransmitted by the medical image acquisition device.

The medical image is an image of the internal tissue of a living body ora part of a living body acquired in a non-invasive manner in medicaltreatment or medical research, for example, an image of the human brain,gut, liver, heart, throat or vagina. The image may be a CT image, an MRIimage, a positron emission tomography image or the like. The living bodyis an independent individual with a life form, for example, a human oran animal. The source domain image may be an image that has beenacquired by the medical image acquisition device and is obtained throughvarious channels, for example, from a database, a network or the like,and may be an image sample obtained by performing annotation withspecific meanings on an image by a professional, or may be an imagesample without any processing.

In step 102, the source domain image and the target domain image aresegmented by using a generative network in a first generativeadversarial network to respectively determine a first source domainsegmentation loss and a first target domain segmentation loss. In anembodiment, step 102 includes segmenting each of the source domainimages and the target domain images by using a generative network in afirst generative adversarial network to respectively determine firstsource domain segmentation losses and first target domain segmentationlosses.

The structure and the parameters of the first generative adversarialnetwork may be set and adjusted according to an actual case. Forexample, the generative network in the first generative adversarialnetwork may use DeepLabv2 with a residual network 101 (ResNet 101) beingthe main framework as a basic model to implement a preliminarysegmentation result. In addition, a spatial pyramid (Atrous SpatialPyramid Pooling, ASPP) structure is added to enrich multi-scaleinformation of a feature map. To enhance the feature expressioncapability of the network, a Dual Attention Network (DANet)-basedattention mechanism is provided, to learn how to capture contextdependence between a pixel and a feature layer channel, and connect anoutput of an attention module and an output of the spatial pyramidstructure to generate a final spatial segmentation feature. Adiscriminative network in the first generative adversarial network mayuse a multilayer fully convolutional network to integrate segmentationprobabilities of the source domain image and the target domain imageinto adversarial learning. One Leaky Rectified Linear Unit (ReLU)activation function layer may be added to each of all convolutionallayers except the last layer, to eventually output a 2D result of asingle channel. 0 and 1 respectively represent a source domain and atarget domain.

For example, feature extraction may be specifically performed on thesource domain image and the target domain image by using the generativenetwork in the first generative adversarial network to respectivelyobtain feature information of a first source domain image and featureinformation of a first target domain image, target segmentation (i.e.,segmentation) is performed on the source domain image based on thefeature information of the first source domain image to determine thefirst source domain segmentation loss, and target segmentation isperformed on the target domain image based on the feature information ofthe first target domain image to determine the first target domainsegmentation loss.

The first source domain segmentation loss may be determined in aplurality of manners. For example, a weighted graph (distance map) of anadversarial noise label may be introduced into the source domain image.Because medical labels have greatly varied marks at a boundary positionof a target area, to prevent a network from fitting a noise label, a newanti-noise segmentation loss is provided, useful pixel level informationis learned from a noise label, and an area with noise at an edge isfiltered out.

For example, the source domain image includes a noisy image and anoiseless image, and target segmentation may be specifically performedon the noisy image in the source domain image based on the featureinformation of the first source domain image to obtain a first noisesegmentation probability; a weighted graph of the noisy image in thesource domain image is obtained; a first noise segmentation loss isobtained according to the first noise segmentation probability and theweighted graph of the noisy image; target segmentation is performed onthe noiseless image in the source domain image based on the featureinformation of the first source domain image to obtain a first noiselesssegmentation probability; a first noiseless segmentation loss isobtained according to the first noiseless segmentation probability and alabeling result of the noiseless image; and the first source domainsegmentation loss is determined based on the first noise segmentationloss and the first noiseless segmentation loss. A first noisesegmentation result is generated according to the first noisesegmentation probability.

Specific formulas of the first noise segmentation loss may be asfollows:

$\begin{matrix}{{{L_{noise}( {p,y} )} = {1 - {\lambda_{1}{\sum\limits_{i = 1}^{k \times w \times c}{( {W( y_{i} )} ){\log( p_{i} )}}}} - {\lambda_{2}\frac{2{\sum\limits_{i = 1}^{h \times w \times c}{( {W( y_{i} )} )p_{i}}}}{{W( y_{i} )}_{i}^{2} + p_{i}^{2}}}}},{and}} & (1) \\{{W( y_{i} )} = \{ {\begin{matrix}{{w_{c} + {w_{0} \times {\exp( {- \frac{( {\max_{dis}{- {d( y_{i} )}}} )^{2}}{2 \times \delta^{2}}} )}}},\ {{{if}\mspace{14mu}{dice}_{co}} \geq \mu}} \\{y_{i},\ {otherwise}}\end{matrix}.} } & (2)\end{matrix}$

In Formula (1), h×w×c respectively represent a length, a width, and aclass of image data, λ₁ and λ₂ are weight coefficients of two losses,W(y_(i)) represents a weighted graph, the second term of the formula isbased on a cross entropy loss, and the third term is based on a diceloss. In Formula (2), w_(c) represents that a weighted weight value forbalancing a class is also a weight coefficient. For each noise labely_(i), a distance d(y_(i)) between a pixel on a label and a closestboundary is calculated, and a maximum value max_(dis) of the distanced(y_(i)) is acquired from a class level area. When two networks exchangeclean data respectively considered having a small loss, dice_(co) ofpredicted values of clean data by the two networks is calculated. Whendice_(co) is greater than a threshold μ, it indicates that the twonetworks have a disagreement about the sample, and the sample isconsidered as a noisy sample (noisy data). The anti-noise segmentationloss is added to improve learning of the noisy sample, and a lossL_(noise) of the networks is calculated, or otherwise, an originalmanner of a cross entropy and a dice loss is kept to calculate a loss.For a weight mapping W(y_(i)), the center of each inter-class area has arelatively large weight, and a position closer to a boundary has asmaller weight. L_(noise) allows the networks to capture a criticalposition at the center, and filters out differences on the boundaryunder various noise labels.

For a target domain dataset, there is no pixel level semantic label.Therefore, the entire task may be considered as an unsupervised imagesegmentation problem. In this application, in a manner of adding“self-supervision” information, a segmentation result of the targetdomain image is used to generate a pixel level pseudo label, and thepseudo label is applied to a next training stage. In a segmentationprobability result of the target domain image, for any pixel, if aprediction confidence level of a class is greater than a confidencethreshold, a pseudo label of a corresponding class is generated at theposition of the pixel. An adaptive setting manner is used for theconfidence threshold. Each class in the target domain image and eachpseudo label in each sample are sorted. A pixel with the highest classlevel prediction confidence level and image level prediction confidencelevel is adaptively selected, to generate a pixel level pseudo label ascross supervision information of the next training stage. To ensure theaccuracy of the generated pseudo label, an “easy to difficult” strategyis used, that is, a model is iteratively trained, to continuouslygenerate a more accurate pseudo label.

For example, target segmentation may be specifically performed on thetarget domain image based on the feature information of the targetdomain image to obtain a first target domain segmentation probability; afirst target domain segmentation result is generated according to thefirst target domain segmentation probability; and the first targetdomain segmentation loss is obtained according to the first targetdomain segmentation result and the target domain image.

Next, a first source domain segmentation probability P_(S) and a firsttarget domain segmentation probability P_(T) of a segmentation resultoutputted by the generative network in the first generative adversarialnetwork are simultaneously inputted into the discriminative network inthe first generative adversarial network, an adversarial loss L_(D) iscalculated by using an information entropy result generated by P_(T),and at the same time a parameter of the discriminative network isupdated by maximising the adversarial loss. Subsequently, an errorgenerated by an adversarial loss function is also transmitted back tothe generative network, and a parameter of a segmentation network isupdated by minimizing the adversarial loss. The objective is to makesegmentation results predicted for the source domain image and thetarget domain image by the generative network increasingly similar, toimplement field adaptivity.

For example, after a first source domain segmentation result and thefirst target domain segmentation result are obtained, the first sourcedomain segmentation result and the first target domain segmentationresult may be specifically discriminated by using the discriminativenetwork in the first generative adversarial network to obtain a firstdiscrimination result; and the first generative adversarial network istrained according to the first source domain segmentation result, thefirst target domain segmentation result, and the first discriminationresult to obtain the first generative adversarial network aftertraining.

The first source domain segmentation result and the first target domainsegmentation result may be determined in a plurality of manners. Forexample, information entropy of the first target domain image may becalculated; and the first discrimination result is obtained by using thediscriminative network in the first generative adversarial network andaccording to the first source domain segmentation result, the firsttarget domain segmentation result, and the first target domain image.

There may be a plurality of manners of training the first generativeadversarial network according to the first source domain segmentationresult, the first target domain segmentation result, and the firstdiscrimination result. For example, the first source domain segmentationloss may be specifically obtained according to the first source domainsegmentation result and a labeling result of the first source domainimage; the first target domain segmentation loss is obtained accordingto the first target domain segmentation result and the first targetdomain image; a first discrimination loss of the discriminative networkis obtained according to the first source domain segmentation result andthe first target domain segmentation result; and the first generativeadversarial network is trained according to the first source domainsegmentation loss, the first target domain segmentation loss, and thefirst discrimination loss to obtain the first generative adversarialnetwork after training.

There may be a plurality of manners of training the first generativeadversarial network according to the first source domain segmentationloss, the first target domain segmentation loss, and the firstdiscrimination loss. For example, a minimal adversarial loss of thefirst generative adversarial network may be specifically built accordingto the first source domain segmentation loss and the first target domainsegmentation loss; a maximal adversarial loss of the first generativeadversarial network is built according to the first discrimination loss;and iterative training is performed on the first generative adversarialnetwork based on the minimal adversarial loss and the maximaladversarial loss to obtain the first generative adversarial networkafter training.

Specific calculation formulas for minimizing the adversarial loss andmaximizing the adversarial loss (that is, the entire target function isoptimized through maximization and minimization) are as follows:

$\begin{matrix}{{{\min\limits_{G}\max\limits_{D}L_{seg}} + L_{D}}.} & (3)\end{matrix}$

For the source domain image X_(S) and the target domain image X_(T) ,Y_(S) is a label of the source domain image, Y _(T) is a pseudo labelgenerated for the target domain image during training, L_(seg) is a lossfunction P=G(X)ϵR^(H×W×C) of the entire segmentation network (that is,the generative network), and a segmentation loss of the segmentationnetwork is:

$\begin{matrix}{L_{seg} = {{L_{seg}^{S}( {X_{S},Y_{S}} )} + {{L_{seg}^{T}( {X_{T},{\overset{\_}{Y}}_{T}} )}.}}} & (4)\end{matrix}$

A source domain segmentation loss L_(seg) ^(S) is defined as:

$\begin{matrix}{{L_{seg}^{S} = {{( {1 - \alpha} )L_{clean}} + {\alpha\; L_{noise}}}},{and}} & (5) \\{{L_{clean}( {p,y} )} = {1 - {\lambda_{1}{\sum\limits_{i = 1}^{h \times w \times c}{( y_{i} ){\log( p_{i} )}}}} - {\lambda_{2}{\frac{2{\sum\limits_{i = 1}^{h \times w \times c}{( y_{i} )p_{i}}}}{y_{i}^{2} + p_{i}^{2}}.}}}} & (6)\end{matrix}$

L_(noise) the first noise segmentation loss, L_(clean) is a segmentationloss of data (clean data) with a clean and reliable label, that is, thefirst noiseless segmentation loss, and α is a coefficient for balancingbetween L_(clean) and L_(noise),

The calculation of the adversarial loss of the discriminative networkmay be as follows:

$\begin{matrix}{L_{D} = {\lambda_{adv}{{L_{ad\nu}( {X_{S},X_{T}} )}.}}} & (7)\end{matrix}$

L_(adv) is a parameter for balancing a loss relationship of theadversarial loss during training, and L_(adv) may be represented as:

$\begin{matrix}{ {{L_{adv}( {X_{S},X_{T}} )} = {{- {E\lbrack {\log( {D( {G( X_{S} )} )} )} \rbrack}} - {E\lbrack {{\lambda_{entr}{f( X_{T} )}} + ɛ} )} + {\log( {1 - {D( {G( X_{T} )} )}} )}}} \rbrack.} & (8)\end{matrix}$

λ_(entr) is a weight parameter corresponding to an information entropyresult graph, and ε is added to ensure stable training in the case off(X_(T)). f(X_(T)) is an information entropy calculation result of thetarget domain image, and may be represented as:

$\begin{matrix}{{{f( X_{T} )} = {\sum\limits_{i}^{h \times w \times c}{p_{i}{\log( p_{i} )}}}}.} & (9)\end{matrix}$

An information entropy map is introduced into pixel by pixel predictionof the target domain image, and then, according to the prediction, the“entropy map” is multiplied by an adversarial loss calculated for eachpixel by a discriminator, thereby increasing a loss weight of a pixel (ahigh entropy value) with uncertainty, and reducing a loss weight (a lowentropy value) with certainty. Driven by an entropy map mapping, networklearning is assisted in how to focus on the most representative featurein class.

In step 103, the source domain image and the target domain image aresegmented by using a generative network in a second generativeadversarial network to respectively determine a second source domainsegmentation loss and a second target domain segmentation loss. In anembodiment, step 103 may include segmenting each of the source domainimages and the target domain images by using a generative network in asecond generative adversarial network to respectively determine secondsource domain segmentation losses and second target domain segmentationlosses.

The training of the second generative adversarial network is similar tothat of the first generative adversarial network, where differentstructures and parameters are used. For example, for a second generativeadversarial network N2, a DeepLabv3+ architecture may be used. To reducea quantity of parameters and a calculation cost, a lightweight networkMobileNetV2 is used as a basic model. The network N2 uses the firstconvolutional layer of MobileNetV2 and seven subsequent residual blocksto extract features. Similar to the network N1, an ASPP module issimilarly added to learn underlying features in different receptivefields. ASPP with different dilation rates is used to generatemulti-scale features, and semantic information of different layers isintegrated into feature mapping. The feature mapping is upsampled. Thenconvolution is performed. The foregoing combined feature is connected toa low level feature, to perform fine-grained semantic segmentation.

For example, feature extraction may be performed on the source domainimage and the target domain image by using the generative network in thesecond generative adversarial network to respectively obtain featureinformation of the source domain image and feature information of thetarget domain image, target segmentation is performed on the sourcedomain image based on the feature information of the source domain imageto determine the second source domain segmentation loss, and targetsegmentation is performed on the target domain image based on thefeature information of the target domain image to determine the secondtarget domain segmentation loss.

The second source domain segmentation loss may be determined in aplurality of manners. For example, a weighted graph (distance map) of anadversarial noise label may be introduced into the source domain image.Because medical labels have greatly varied marks at a boundary positionof a target area, to prevent a network from fitting a noise label, a newanti-noise segmentation loss is provided, useful pixel level informationis learned from a noise label, and an area with noise at an edge isfiltered out.

For example, the source domain image includes a noisy image and anoiseless image, and target segmentation may be specifically performedon the noisy image in the source domain image based on the featureinformation of the source domain image to obtain a second noisesegmentation probability; a weighted graph of the noisy image in thesource domain image is obtained; a second noise segmentation loss isobtained according to the second noise segmentation probability and theweighted graph of the noisy image; target segmentation is performed onthe noiseless image in the source domain image based on the featureinformation of the source domain image to obtain a second noiselesssegmentation probability; a second noiseless segmentation loss isobtained according to the second noiseless segmentation probability anda labeling result of the noiseless image; and the second source domainsegmentation loss is determined based on the second noise segmentationloss and the second noiseless segmentation loss.

For a specific calculation manner of the second noise segmentation loss,reference may be made to the foregoing calculation manner of the firstnoise segmentation loss.

For the target domain image, a specific training manner is similar tothat of a preset first generative network, or a manner of adding“self-supervision” information may be added, that is, a segmentationresult of the target domain image is used to generate a pixel levelpseudo label, and the pseudo label is applied to a next training stage.For example, target segmentation may be specifically performed on thetarget domain image based on the feature information of the secondtarget domain image to obtain a second target domain segmentationprobability; a second target domain segmentation result is generatedaccording to the second target domain segmentation probability; and thesecond target domain segmentation loss is obtained according to thesecond target domain segmentation result and the target domain image.

Next, a second source domain segmentation probability P_(S) and a secondtarget domain segmentation probability P_(T) of a segmentation resultoutputted by the generative network in the second generative adversarialnetwork are simultaneously inputted into a discriminative network in thesecond generative adversarial network, an adversarial loss L_(D) iscalculated by using an information entropy result generated by P_(T),and at the same time a parameter of the discriminative network isupdated by maximizing the adversarial loss. Subsequently, an errorgenerated by an adversarial loss function is also transmitted back tothe generative network, and a parameter of a segmentation network isupdated by minimizing the adversarial loss. The objective is to makesegmentation results predicted for the source domain image and thetarget domain image by the generative network increasingly similar, toimplement field adaptivity.

For example, after a second source domain segmentation result and thesecond target domain segmentation result are obtained, the second sourcedomain segmentation result and the second target domain segmentationresult may be specifically discriminated by using the discriminativenetwork in the second generative adversarial network to obtain a seconddiscrimination result; the second generative adversarial network istrained according to the second source domain segmentation result, thesecond target domain segmentation result, and the second discriminationresult to obtain the second generative adversarial network aftertraining.

The second source domain segmentation result and the second targetdomain segmentation result may be determined in a plurality of manners.For example, information entropy of the target domain image may bespecifically calculated; and the second discrimination result isobtained by using the discriminative network in the second generativeadversarial network and according to the second source domainsegmentation result, the second target domain segmentation result, andthe target domain image.

There may be a plurality of manners of training the second generativeadversarial network according to the second source domain segmentationresult, the second target domain segmentation result, and the seconddiscrimination result. For example, the second source domainsegmentation loss may be specifically obtained according to the secondsource domain segmentation result and a labeling result of the secondsource domain image; the second target domain segmentation loss isobtained according to the second target domain segmentation result andthe second target domain image; a second discrimination loss of thediscriminative network is obtained according to the second source domainsegmentation result and the second target domain segmentation result;and the second generative adversarial network is trained according tothe second source domain segmentation loss, the second target domainsegmentation loss, and the second discrimination loss to obtain thesecond generative adversarial network after training.

There may be a plurality of manners of training the second generativeadversarial network according to the second source domain segmentationloss, the second target domain segmentation loss, and the seconddiscrimination loss. For example, a minimal adversarial loss of thesecond generative adversarial network may be specifically builtaccording to the second source domain segmentation loss and the secondtarget domain segmentation loss; a maximal adversarial loss of thesecond generative adversarial network is built according to the seconddiscrimination loss; and iterative training is performed on the secondgenerative adversarial network based on the minimal adversarial loss andthe maximal adversarial loss to obtain the second generative adversarialnetwork after training.

A calculation method of each loss in the second generative adversarialnetwork is similar to that of the first generative adversarial network.Reference may be made to the foregoing description for details.

In step 104, a first source domain image and a second source domainimage are determined according to the first source domain segmentationloss and the second source domain segmentation loss, and a first targetdomain image and a second target domain image according to the firsttarget domain segmentation loss and the second target domainsegmentation loss. In an embodiment, step 104 includes determining afirst source domain image and a second source domain image according tothe first source domain segmentation losses and the second source domainsegmentation losses, and determining a first target domain image and asecond target domain image according to the first target domainsegmentation losses and the second target domain segmentation losses.

During training, in a training manner of cross training, clean sourcedomain image data selected from two different generative networks ineach stage is used to update network parameters step by step. Specifictraining steps are as follows: Step 1: After N times of iteration, eachgenerative adversarial network sorts segmentation losses of allpredicted values, and two networks respectively select small losssamples C₁ and C₂ as clean data. Step 2: Each network sends these usefulsamples to its peer network to perform a next training process, and thenupdates parameters of the convolutional layer. Step 3: Each generativenetwork reselects clean data that is considered optimal currently, andtunes its peer network in a layered manner. Because the two networkshave different structures and learning capabilities, the two networkscan filter errors of different types introduced by noise labels. In thisexchange process, peer networks can supervise each other, therebyreducing training errors caused by noise labels.

For example, the first source domain segmentation losses may bespecifically sorted, a source domain image meeting a preset losscondition is selected according to the sorted first source domainsegmentation losses, and the source domain image is determined as thefirst source domain image; and the second source domain segmentationlosses are sorted, a source domain image meeting the preset losscondition is selected according to the sorted second source domainsegmentation losses, and the source domain image is determined as thesecond source domain image.

The preset loss condition may be, for example, a preset loss threshold.Correspondingly, the source domain image meets a preset condition, forexample, when a source domain segmentation loss of the source domainimage is less than the loss threshold. The preset loss condition may bealternatively that the loss threshold is minimum. Correspondingly, thesource domain image that meets the preset loss condition is a sourcedomain image with the minimum source domain segmentation loss in allsource domain images.

For the target domain image, cross learning is performed on pseudolabels generated by two generative networks for the target domain imagein each stage to update network parameters. Specific training steps areas follows: Step 1: Use results PL₁ and PL₂ of training results of twonetworks in a previous stage for the target domain as pseudo labels.Step 2: Apply the pseudo labels to a training process of another networkin a next stage, to iteratively update network parameters. In eachstage, the segmentation network and the discriminative network aretrained in an alternate update manner. Image data is first inputted intothe segmentation network, and a segmentation loss L_(seg) is calculatedby using a real label of the source domain data and the pseudo label ofthe target domain data, and a parameter of the segmentation network isupdated by minimizing the segmentation loss.

For example, the first generative adversarial network may bespecifically trained according to the first target domain segmentationlosses, and the first target domain image is generated by using atraining result; and the second generative adversarial network istrained according to the second target domain segmentation losses, andthe second target domain image is generated by using a training result.

In step 105, cross training is performed on the first generativeadversarial network and the second generative adversarial network byusing the first source domain image, the first target domain image, thesecond source domain image, and the second target domain image to obtainthe first generative adversarial network after training. In anembodiment, step 105 includes performing cross training on the firstgenerative adversarial network and the second generative adversarialnetwork by using the first source domain image, the first target domainimage, the second source domain image, and the second target domainimage to obtain a trained first generative adversarial network

For example, the second generative adversarial network may be trained byusing the first source domain image and the first target domain image.The first generative adversarial network is trained by using the secondsource domain image and the second target domain image.

For example, the second source domain image and the second target domainimage may be specifically segmented by using the generative network inthe first generative adversarial network to respectively obtain a secondsource domain segmentation result and a second target domainsegmentation result; the second source domain segmentation result andthe second target domain segmentation result are discriminated by usinga discriminative network in the first generative adversarial network toobtain a second discrimination result; and training the first generativeadversarial network according to the second source domain segmentationresult, the second target domain segmentation result, and the seconddiscrimination result to obtain the first generative adversarial networkafter training.

The second source domain segmentation result and the second targetdomain segmentation result may be determined in a plurality of manners.For example, information entropy of the second target domain image maybe calculated; and the second discrimination result is obtained by usingthe discriminative network in the first generative adversarial networkand according to the second source domain segmentation result, thesecond target domain segmentation result, and the second target domainimage.

There may be a plurality of manners of training the first generativeadversarial network according to the second source domain segmentationresult, the second target domain segmentation result, and the seconddiscrimination result. For example, a second source domain segmentationloss may be specifically obtained according to the second source domainsegmentation result and a labeling result of the second source domainimage; a second target domain segmentation loss is obtained according tothe second target domain segmentation result and the second targetdomain image; a second discrimination loss of the discriminative networkis obtained according to the second source domain segmentation resultand the second target domain segmentation result; and the firstgenerative adversarial network is trained according to the second sourcedomain segmentation loss, the second target domain segmentation loss,and the second discrimination loss to obtain the first generativeadversarial network after training.

There may be a plurality of manners of training the first generativeadversarial network according to the second source domain segmentationloss, the second target domain segmentation loss, and the seconddiscrimination loss. For example, a minimal adversarial loss of thefirst generative adversarial network may be specifically built accordingto the second source domain segmentation loss and the second targetdomain segmentation loss; a maximal adversarial loss of the firstgenerative adversarial network is built according to the second targetdiscrimination loss; and iterative training is performed on the firstgenerative adversarial network based on the minimal adversarial loss andthe maximal adversarial loss to obtain the first generative adversarialnetwork after training.

A manner of training the second generative adversarial network by usingthe first source domain image and the first target domain image issimilar to a training manner of the second generative adversarialnetwork. For example, the first source domain image and the first targetdomain image may be specifically segmented by using the generativenetwork in the second generative adversarial network to respectivelyobtain a first source domain segmentation result and a first targetdomain segmentation result; the first source domain segmentation resultand the first target domain segmentation result are discriminated byusing a discriminative network in the second generative adversarialnetwork to obtain a first discrimination result; and the secondgenerative adversarial network is trained according to the first sourcedomain segmentation result, the first target domain segmentation result,and the first discrimination result to obtain the second generativeadversarial network after training.

In some embodiments, the first source domain segmentation result and thefirst target domain segmentation result are discriminated by using thediscriminative network in the second generative adversarial network,information entropy of the first target domain image may be specificallycalculated, and the first discrimination result is obtained by using thediscriminative network in the second generative adversarial network andaccording to the first source domain segmentation result, the firsttarget domain segmentation result, and the first target domain image.

In some embodiments, the second generative adversarial network istrained according to the first source domain segmentation result, thefirst target domain segmentation result, and the first discriminationresult. A first source domain segmentation loss may be specificallyobtained according to the first source domain segmentation result and alabeling result of the first source domain image; a first target domainsegmentation loss is obtained according to the first target domainsegmentation result and the first target domain image; a firstdiscrimination loss of the discriminative network is obtained accordingto the first source domain segmentation result and the first targetdomain segmentation result; and the second generative adversarialnetwork is trained according to the first source domain segmentationloss, the first target domain segmentation loss, and the firstdiscrimination loss to obtain the second generative adversarial networkafter training.

In some embodiments, a minimal adversarial loss of the second generativeadversarial network may be specifically built according to the firstsource domain segmentation loss and the first target domain segmentationloss; a maximal adversarial loss of the second generative adversarialnetwork is built according to the first discrimination loss; anditerative training is performed on the second generative adversarialnetwork based on the minimal adversarial loss and the maximaladversarial loss to obtain the second generative adversarial networkafter training.

In step 106, a to-be-segmented image is segmented based on thegenerative network in the first generative adversarial network aftertraining to obtain a segmentation result. In an embodiment, step 106includes segmenting a to-be-segmented image based on the generativenetwork in the trained first generative adversarial network to obtain asegmentation result.

For example, feature extraction may be specifically performed on theto-be-segmented image based on the generative network in the firstgenerative adversarial network after training to obtain featureinformation of the to-be-segmented image. Target segmentation isperformed on the to-be-segmented image based on the feature informationof the to-be-segmented image to obtain a segmentation predictionprobability of the to-be-segmented image, and a segmentation result ofthe to-be-segmented image is generated according to the segmentationprediction probability.

The to-be-segmented image is an image that needs to be segmented, forexample, a medical image (for example, a heart image or a lung image) orsome ordinary images (for example, a person image and an object image).For example, when the to-be-segmented image is a medical image, themedical image may be obtained by performing image acquisition on thetissue of a living body by the medical image acquisition device such asa CT device or an MRI device, for example, the human brain, gut, liver,heart, throat or vagina, to provide the medical image to a medical imagedetection apparatus. That is, the medical image detection apparatus mayspecifically receive the to-be-segmented image transmitted by themedical image acquisition device.

As can be seen from above, in this embodiment of this application,target domain images and source domain images that are labeled withtarget information are obtained first, then the source domain images andthe target domain images are segmented by using a generative network ina first generative adversarial network to respectively determine a firstsource domain segmentation loss and a first target domain segmentationloss, the source domain image and the target domain image are segmentedby using a generative network in a second generative adversarial networkto respectively determine a second source domain segmentation loss and asecond target domain segmentation loss, next, a first source domainimage and a second source domain image are determined according to thefirst source domain segmentation loss and the second source domainsegmentation loss, a first target domain image and a second targetdomain image are determined according to the first target domainsegmentation loss and the second target domain segmentation loss, thencross training is performed on the first generative adversarial networkand the second generative adversarial network by using the first sourcedomain image, the first target domain image, the second source domainimage, and the second target domain image to obtain the first generativeadversarial network after training, and a to-be-segmented image is thensegmented based on the generative network in the first generativeadversarial network after training to obtain a segmentation result. Thesolution provides an unsupervised robust segmentation method based on adomain adaptive strategy for the phenomenon that there is noise inlabels of data and a distribution difference between source domain andtarget domain datasets. Therefore, in the manner of mutual learning andmutual supervision between two models, labels with noise and anunsupervised image segmentation task are resolved, thereby effectivelyimproving the accuracy of image segmentation. According to the methoddescribed in the previous embodiment, accurate segmentation of an opticcup and an optic disc of glaucoma is used as an example below forfurther detailed description.

To ensure that the algorithm can actually assist in clinical diagnosis,the accuracy of image segmentation needs to be improved. Embodiments ofthis application provide a robust unsupervised field adaptivesegmentation method based on noise label data. A feature structure on adataset with labels can be learned, knowledge is transferred to a newdataset, and relatively accurate image segmentation is provided for anew dataset without labels, thereby effectively improving thegeneralization performance of a deep network on other datasets.

The unsupervised field adaptive training method in this embodiment ofthis application may train a generative adversarial network including animage segmentation network (as a generative network) in a fieldadversarial manner. Next, the generative network in the trainedgenerative adversarial network is used to segment an unlabeledto-be-segmented image. In this embodiment, an example in which the imagesegmentation apparatus is specifically integrated in an electronicdevice is used for description.

As shown in FIG. 2a , an embodiment of this application provides animage segmentation method. A specific procedure may be as follows.

In step 201, an electronic device obtains a target domain image and asource domain image that is labeled with target information. In anembodiment, the electronic device obtains plural target domain imagesand plural source domain images that are labeled with targetinformation.

Specifically, in an adaptive segmentation task of two fundus imagedatasets, datasets REFUGE and Drishti-GS are used. Because training setsand validation sets (or test sets) of the datasets are photographed bydifferent acquisition devices, there are differences in color, texture,and the like of images. The training set of the REFUGE dataset is usedas the source domain training set. The validation set of the REFUGEdataset and the validation set of the Drishti-GS dataset are used astarget domain training sets. The test set of the REFUGE dataset and thetest set of the Drishti-GS dataset are used as target domain test sets.For the REFUGE dataset, the training set includes 400 images, an imagesize is 2124×2056, the validation set includes 300 images, the test setincludes 100 images, and an image size is 1634×1634. For the Drishti-GSdataset, the validation set includes 50 images, the test set includes 51images, and an image size is 2047×1759.

This application provides an unsupervised robust segmentation methodbased on a domain adaptive strategy for the phenomenon that there is adistribution difference between source domain and target domaindatasets. Therefore, in the manner of mutual learning and mutualsupervision between two models, an unsupervised image segmentation taskis effectively resolved. A framework of the robust segmentation methodshown in FIG. 2b is formed by two generative adversarial networks, thatis, N1 and N2. N1 includes a generative network (also referred to as asegmentation network) S1 and a discriminative network D1. N2 includes agenerative network (also referred to as a segmentation network) S2 and adiscriminative network D2. The two networks have different structuresand parameters. Due to the differences in the structures and parametersof the two networks, the two networks may generate different decisionboundaries, that is, have different learning capabilities, therebypromoting peer-review between networks. The peer-review is a strategyfor mutual review of two networks. The two networks exchange small lossdata and pseudo labels, to improve the performance of the networks.

In step 202, the electronic device segments the source domain image andthe target domain image by using a generative network in a firstgenerative adversarial network to respectively determine a first sourcedomain segmentation loss and a first target domain segmentation loss. Inan embodiment, step 202 includes the electronic device segmenting one ormore of the source domain images and the target domain images by using agenerative network in a first generative adversarial network torespectively determine first source domain segmentation losses and firsttarget domain segmentation losses.

For example, as shown in FIG. 2c , the generative network in the firstgenerative adversarial network may use DeepLabv2 with ResNet101 as themain framework as the basic model, to implement a preliminarysegmentation result. In addition, an ASPP structure is added, andmulti-scale information of a feature map is enriched. To enhance thefeature expression capability of the network, a DANet-based attentionmechanism is provided, to learn how to capture context dependencebetween a pixel and a feature layer channel, and connect an output of anattention module and an output of the spatial pyramid structure togenerate a final spatial segmentation feature.

For example, each source domain image includes a noisy image and anoiseless image, and a weighted graph of an adversarial noise label maybe introduced into the noisy image in the source domain image. Becausemedical labels have greatly varied marks at a boundary position of atarget area, to prevent a network from fitting a noise label, a newanti-noise segmentation loss is provided, useful pixel level informationis learned from a noise label, and an area with noise at an edge isfiltered out.

For example, the electronic device may specifically perform featureextraction on the source domain image by using the generative network inthe first generative adversarial network to obtain feature informationof the source domain image, perform target segmentation on the sourcedomain image based on the feature information of the source domain imageto obtain a first noise segmentation probability, obtain a weightedgraph of the noisy image in the source domain image, and obtain a firstnoise segmentation loss according to the first noise segmentationprobability and the weighted graph of the noisy image; perform targetsegmentation on the noiseless image in the source domain image based onthe feature information of the source domain image to obtain a firstnoiseless segmentation probability; and obtain a first noiselesssegmentation loss according to the first noiseless segmentationprobability and a labeling result of the noiseless image, determine thefirst source domain segmentation loss based on the first noisesegmentation loss and the first noiseless segmentation loss, andgenerate a first noise segmentation result according to the first noisesegmentation probability. For a calculation manner of the first noisesegmentation loss, reference may be made to the foregoing embodimentsfor details.

For example, the electronic device may specifically perform featureextraction on the target domain image by using the generative network inthe first generative adversarial network to obtain feature informationof the target domain image, perform target segmentation on the targetdomain image based on the feature information of the target domain imageto obtain a first target domain segmentation probability; generate afirst target domain segmentation result according to the first targetdomain segmentation probability; and obtain the first target domainsegmentation loss according to the first target domain segmentationresult and the target domain image.

Next, a first source domain segmentation probability P_(S) and a firsttarget domain segmentation probability P_(T) of a segmentation resultoutputted by the generative network in the first generative adversarialnetwork are simultaneously inputted into a discriminative network in thefirst generative adversarial network, an adversarial loss L_(D) iscalculated by using an information entropy result generated by P_(T),and at the same time a parameter of the discriminative network isupdated by maximizing the adversarial loss. Subsequently, an errorgenerated by an adversarial loss function is also transmitted back tothe generative network, and a parameter of a segmentation network isupdated by minimizing the adversarial loss. The objective is to makesegmentation results predicted for the source domain image and thetarget domain image by the generative network increasingly similar, toimplement field adaptivity.

For example, the discriminative network in the first generativeadversarial network may use a five-layer fully convolutional network tointegrate segmentation probabilities of the source domain and the targetdomain into adversarial learning. A kernel size of each convolutionallayer of a network model is 4, a stride is 2, and a padding is 1. Inaddition, one Leaky ReLU activation function layer is added to each ofall convolutional layers except the last layer, to eventually output a2D result of a single channel. 0 and 1 respectively represent a sourcedomain and a target domain.

For example, after a first source domain segmentation result and thefirst target domain segmentation result are obtained, informationentropy of the first target domain image may be specifically calculated;a first discrimination result is obtained by using the discriminativenetwork in the first generative adversarial network and according to thefirst source domain segmentation result, the first target domainsegmentation result, and the first target domain image, and then thefirst source domain segmentation loss is obtained according to the firstsource domain segmentation result and a labeling result of the firstsource domain image; the first target domain segmentation loss isobtained according to the first target domain segmentation result andthe first target domain image; a first discrimination loss of thediscriminative network is obtained according to the first source domainsegmentation result and the first target domain segmentation result; anda minimal adversarial loss of the first generative adversarial networkis built according to the first source domain segmentation loss and thefirst target domain segmentation loss; a maximal adversarial loss of thefirst generative adversarial network is built according to the firstdiscrimination loss; and iterative training is performed on the firstgenerative adversarial network based on the minimal adversarial loss andthe maximal adversarial loss to obtain the first generative adversarialnetwork after training.

For a specific calculation manner of minimizing the adversarial loss andmaximizing the adversarial loss (that is, the entire target function isoptimized through maximization and minimization), reference may be madeto the foregoing embodiments for details.

In step 203, the electronic device segments the source domain image andthe target domain image by using a generative network in a secondgenerative adversarial network to respectively determine a second sourcedomain segmentation loss and a second target domain segmentation loss.In an embodiment, step 203 includes the electronic device segmenting oneor more of the source domain images and the target domain images byusing a generative network in a second generative adversarial network torespectively determine second source domain segmentation losses andsecond target domain segmentation losses.

The training of the second generative adversarial network is similar tothat of the first generative adversarial network, where differentstructures and parameters are used. For example, for a second generativeadversarial network N2, a DeepLabv3+ architecture may be used. To reducea quantity of parameters and a calculation cost, a lightweight networkMobileNetV2 may be used as a basic model. The second generativeadversarial network N2 uses the first convolutional layer of MobileNetV2and seven subsequent residual blocks to extract features. A stride ofthe first convolutional layer and two subsequent residual blocks may beset to 2, and a stride of the remaining blocks is set to 1. A totaldownsampling rate of the second generative adversarial network is 8.Similar to the first generative adversarial network Ni, an ASPP moduleis similarly added to learn underlying features in different receptivefields. ASPP with different dilation rates is used to generatemulti-scale features, and semantic information of different layers isintegrated into feature mapping. The feature mapping is upsampled. Then1×1 convolution is performed. The foregoing combined feature isconnected to a low level feature, to perform fine-grained semanticsegmentation.

For example, the electronic device may specifically perform featureextraction on the source domain image by using the generative network inthe second generative adversarial network to obtain feature informationof the source domain image, and perform target segmentation on thesource domain image based on the feature information of the sourcedomain image to obtain a second noise segmentation probability; obtain aweighted graph of the noisy image in the source domain image; obtain asecond noise segmentation loss according to the second noisesegmentation probability and the weighted graph of the noisy image;perform target segmentation on the noiseless image in the source domainimage based on the feature information of the source domain image toobtain a second noiseless segmentation probability; obtain a secondnoiseless segmentation loss according to the second noiselesssegmentation probability and a labeling result of the noiseless image;and determine the second source domain segmentation loss based on thesecond noise segmentation loss and the second noiseless segmentationloss.

For a specific calculation manner of the second noise segmentation loss,reference may be made to the foregoing calculation manner of the firstnoise segmentation loss.

For the target domain image, a specific training manner is similar tothat of a preset first generative network, or a manner of adding“self-supervision” information may be added. That is, a segmentationresult of the target domain image is used to generate a pixel levelpseudo label, and the pseudo label is applied to a next training stage.For example, the electronic device may specifically perform featureextraction on the target domain image by using the generative network inthe second generative adversarial network to obtain feature informationof a second target domain image, perform target segmentation on thetarget domain image based on the feature information of the secondtarget domain image to obtain a second target domain segmentationprobability; generate a second target domain segmentation resultaccording to the second target domain segmentation probability; andobtain the second target domain segmentation loss according to thesecond target domain segmentation result and the target domain image.

Next, a second source domain segmentation probability P_(S) and a secondtarget domain segmentation probability P_(T) of a segmentation resultoutputted by the generative network in the second generative adversarialnetwork are simultaneously inputted into a discriminative network in thesecond generative adversarial network, an adversarial loss L_(D) iscalculated by using an information entropy result generated by P_(T),and at the same time a parameter of the discriminative network isupdated by maximizing the adversarial loss. Subsequently, an errorgenerated by an adversarial loss function is also transmitted back tothe generative network, and a parameter of a segmentation network isupdated by minimizing the adversarial loss. The objective is to makesegmentation results predicted for the source domain image and thetarget domain image by the generative network increasingly similar, toimplement field adaptivity. In a process of optimizing a networkparameter, in this embodiment, a Stochastic Gradient Descent (SOD)algorithm is used to optimize and train the segmentation network, anadaptive momentum stochastic optimization (Adam) algorithm is used tooptimize and train the discriminative network, and initial learningrates of the segmentation network and the discriminative network arerespectively 2.5×10⁻⁴ and 1×10⁻⁴.

For example, after a second source domain segmentation result and thesecond target domain segmentation result are obtained, the electronicdevice may specifically calculate information entropy of the targetdomain image; obtain the second discrimination result by using thediscriminative network in the second generative adversarial network andaccording to the second source domain segmentation result, the secondtarget domain segmentation result, and the target domain image; thenobtain the second source domain segmentation loss according to thesecond source domain segmentation result and a labeling result of thesource domain image; obtain the second target domain segmentation lossaccording to the second target domain segmentation result and the targetdomain image; obtain a second discrimination loss of the discriminativenetwork according to the second source domain segmentation result andthe second target domain segmentation result; next, build a minimaladversarial loss of the second generative adversarial network accordingto the second source domain segmentation loss and the second targetdomain segmentation loss; build a maximal adversarial loss of the secondgenerative adversarial network according to the second discriminationloss; and perform iterative training on the second generativeadversarial network based on the minimal adversarial loss and themaximal adversarial loss to obtain the second generative adversarialnetwork after training.

A calculation method of each loss in the second generative adversarialnetwork is similar to that of the first generative adversarial network.Reference may be made to the foregoing description for details.

In step 204, the electronic device determines a first source domaintarget image and a second source domain target image according to thefirst source domain segmentation loss and the second source domainsegmentation loss. In an embodiment, step 204 includes the electronicdevice determining a first source domain image and a second sourcedomain image according to the first source domain segmentation lossesand the second source domain segmentation losses.

For example, the electronic device may specifically sort the firstsource domain segmentation losses, select a source domain image meetinga preset loss condition according to the sorted first source domainsegmentation losses, and determine the source domain image as the firstsource domain image (that is, a first source domain clean image); andsort the second source domain segmentation losses, select a sourcedomain image meeting the preset loss condition according to the sortedsecond source domain segmentation losses, and determine the sourcedomain image as the second source domain image (that is, a second sourcedomain clean image). Each generative network sends these clean images toits peer network to perform a next training process, to updateparameters of the convolutional layer. Then each generative networkre-selects clean data that is considered optimal currently, and tunesits peer network in a layered manner. In this exchange process, peernetworks can supervise each other, thereby reducing training errorscaused by noise labels.

In step 205, the electronic device determines a first target domaintarget image and a second target domain target image according to thefirst target domain segmentation loss and the second target domainsegmentation loss. In an embodiment, step 205 includes the electronicdevice determining a first target domain image and a second targetdomain image according to the first target domain segmentation lossesand the second target domain segmentation losses.

For example, the electronic device may specifically train the firstgenerative adversarial network according to the first target domainsegmentation losses, and generate the first target domain image by usinga training result (that is, a pixel level pseudo label of the firsttarget domain image); and train the second generative adversarialnetwork according to the second target domain segmentation losses, andgenerate the second target domain image by using a training result (thatis, a pixel level pseudo label of the second target domain image). Thesepseudo labels are then applied to a training process of another networkin a next stage, to iteratively update network parameters. In eachstage, the segmentation network and the discriminative network aretrained in an alternate update manner.

In step 206, the electronic device trains the first generativeadversarial network by using the second source domain target image andthe second target domain target image to obtain the first generativeadversarial network after training. In an embodiment, step 206 includesthe electronic device training the first generative adversarial networkaccording to the second source domain segmentation result, the secondtarget domain segmentation result, and the second discrimination resultto obtain the trained first generative adversarial network.

For example, the second source domain image and the second target domainimage may be specifically segmented by using the generative network inthe first generative adversarial network to respectively obtain a secondsource domain segmentation result and a second target domainsegmentation result; next, information entropy of the second targetdomain image is calculated; and a second target discrimination result isobtained by using the discriminative network in the first generativeadversarial network and according to the second source domainsegmentation result, the second target domain segmentation result, andthe second target domain image; then a second source domain segmentationloss is obtained according to the second source domain segmentationresult and a labeling result of the second source domain image; a secondtarget domain segmentation loss is obtained according to the secondtarget domain segmentation result and the second target domain image; asecond discrimination loss of the discriminative network is obtainedaccording to the second source domain segmentation result and the secondtarget domain segmentation result; then, a minimal adversarial loss ofthe first generative adversarial network is built according to thesecond source domain segmentation loss and the second target domainsegmentation loss; a maximal adversarial loss of the first generativeadversarial network is built according to the second discriminationloss; and iterative training is performed on the first generativeadversarial network based on the minimal adversarial loss and themaximal adversarial loss to obtain the first generative adversarialnetwork after training.

In step 207, the electronic device trains the second generativeadversarial network by using the first source domain target image andthe first target domain target image to obtain the second generativeadversarial network after training. In an embodiment, step 207 includesthe electronic device training the second generative adversarial networkaccording to the first source domain segmentation result, the firsttarget domain segmentation result, and the first discrimination resultto obtain a trained second generative adversarial network.

For example, the electronic device may specifically segment the firstsource domain image and the first target domain image by using thegenerative network in the second generative adversarial network torespectively obtain a first source domain segmentation result and afirst target domain segmentation result; calculate information entropyof the first target domain image, and obtain the first targetdiscrimination result by using the discriminative network in the secondgenerative adversarial network and according to the first source domainsegmentation result, the first target domain segmentation result, andthe first target domain image; next, obtain a first source domainsegmentation loss according to the first source domain segmentationresult and a labeling result of the first source domain image; obtain afirst target domain segmentation loss according to the first targetdomain segmentation result and the first target domain image; obtain afirst discrimination loss of the discriminative network according to thefirst source domain segmentation result and the first target domainsegmentation result; then build a minimal adversarial loss of the secondgenerative adversarial network according to the first source domainsegmentation loss and the first target domain segmentation loss; build amaximal adversarial loss of the second generative adversarial networkaccording to the first discrimination loss; perform iterative trainingon the second generative adversarial network based on the minimaladversarial loss and the maximal adversarial loss to obtain the secondgenerative adversarial network after training; and then train the secondgenerative adversarial network according to the first source domainsegmentation loss, the first target domain segmentation loss, and thefirst discrimination loss to obtain the second generative adversarialnetwork after training.

In step 208, the electronic device segments a to-be-segmented imagebased on the generative network in the first generative adversarialnetwork after training to obtain a segmentation result. In anembodiment, step 208 includes the electronic device segmenting ato-be-segmented image based on the generative network in the trainedfirst generative adversarial network to obtain a segmentation result.

For example, the electronic device may specifically receive a fundusimage acquired by a medical imaging device, and then feature extractionis performed on the fundus image based on the generative network in thefirst generative adversarial network after training, to obtain featureinformation of the fundus image. Target segmentation is performed on thefundus image based on the feature information of the fundus image toobtain a segmentation prediction probability of the fundus image, and asegmentation result of the fundus image is generated according to thesegmentation prediction probability.

In addition, to validate the effect of the segmentation solutionprovided in this embodiment of this application, experimental results ofthe technologies provided in this application are compared with those ofsome related algorithms, and experimental results based on differentnoise grade tasks are separately shown in Table 1 and Table 2. Table 1shows experimental results of a low noise level from the training set ofREFUGE and the validation set of REFUGE. Table 2 shows experimentalresults of a low noise level from the training set of REFUGE and thevalidation set of REFUGE. In the solution, experimental results inREFUGE and Drishti-GS datasets are shown in FIG. 2d . BDL is aself-supervised learning-based bidirectional learning method, and isused for mitigating a domain shift problem, to learn a bettersegmentation model. pOSAL is OD and OC segmentation tasks of RetinalFundus Glaucoma Challenge. BEAL provides an adversarial learning methodbased on edge and entropy information. In two tasks, experimentalresults of the method provided in this application on different noisegrades and noise levels, where DI is a measurement indicator of asegmentation result, is used for calculating a similarity degree betweena label Y and a predicted value P , and is represented as:

${DI} = {\frac{2{{p\bigcap Y}}}{{p} + {Y}}.}$

Table 1 shows experimental results of a low noise level from thetraining set of REFUGE and the validation set of REFUGE

Noise This Pre- propor- BDL pOSAL BEAL solution training tion DI_(disc)DI_(cup) DI_(disc) DI_(cup) DI_(disc) DI_(cup) DI_(disc) DI_(cup) Yes 00.946 0.874 0.949 0.887 0.933 0.831 0.953 0.894 0.1 0.948 0.887 0.9540.880 0.931 0.820 0.951 0.893 0.3 0.949 0.881 0.953 0.865 0.915 0.7980.948 0.893 0.5 0.949 0.890 0.949 0.859 0.902 0.805 0.954 0.896 0.70.945 0.888 0.946 0.852 0.883 0.801 0.947 0.890 0.9 0.942 0.868 0.9450.858 0.877 0.805 0.953 0.894 No 0.1 0.942 0.867 0.941 0.879 0.927 0.7710.947 0.884 0.3 0.933 0.867 0.941 0.861 0.905 0.767 0.945 0.871 0.50.932 0.860 0.940 0.850 0.878 0.758 0.948 0.867 0.7 0.928 0.851 0.9360.845 0.873 0.708 0.946 0.854 0.9 0.906 0.765 0.925 0.836 0.828 0.6910.941 0.846Table 2 shows experimental results of a low noise level from thetraining set of REFUGE and the validation set of REFUGE

Noise This Pre- propor- BDL pOSAL BEAL solution training tion DI_(disc)DI_(cup) DI_(disc) DI_(cup) DI_(disc) DI_(cup) DI_(disc) DI_(cup) Yes0.1 0.947 0.833 0.947 0.865 0.924 0.818 0.951 0.890 0.3 0.916 0.7990.869 0.741 0.912 0.782 0.947 0.851 0.5 0.916 0.796 0.858 0.756 0.8910.772 0.939 0.856 0.7 0.910 0.757 0.853 0.751 0.807 0.719 0.933 0.8450.9 0.902 0.743 0.845 0.760 0.759 0.669 0.930 0.836 No 0.1 0.895 0.7590.882 0.787 0.912 0.738 0.945 0.888 0.3 0.859 0.761 0.849 0.769 0.8870.689 0.941 0.848 0.5 0.858 0.756 0.839 0.745 0.869 0.691 0.938 0.8470.7 0.853 0.751 0.804 0.729 0.799 0.645 0.932 0.838 0.9 0.847 0.6600.790 0.720 0.778 0.532 0.929 0.811

As can be seen from above, in this embodiment of this application,target domain images and source domain images that are labeled withtarget information are obtained first, then the source domain images andthe target domain images are segmented by using a generative network ina first generative adversarial network to respectively determine firstsource domain segmentation losses and first target domain segmentationlosses, the source domain images and the target domain images aresegmented by using a generative network in a second generativeadversarial network to respectively determine second source domainsegmentation losses and second target domain segmentation losses, next,a first source domain image and a second source domain image aredetermined according to the first source domain segmentation losses andthe second source domain segmentation losses, a first target domainimage and a second target domain image are determined according to thefirst target domain segmentation losses and the second target domainsegmentation losses, then cross training is performed on the firstgenerative adversarial network and the second generative adversarialnetwork by using the first source domain image, the first target domainimage, the second source domain image, and the second target domainimage to obtain the first generative adversarial network after training,and a to-be-segmented image is then segmented based on the generativenetwork in the first generative adversarial network after training toobtain a segmentation result. The solution provides an unsupervisedrobust segmentation method based on a domain adaptive strategy for thephenomenon that there is noise in labels of data and a distributiondifference between source domain and target domain datasets. Therefore,in the manner of mutual learning and mutual supervision between twomodels, labels with noise and an unsupervised image segmentation taskare resolved, thereby effectively improving the accuracy of imagesegmentation.

The steps in the embodiments of this application are not necessarilyperformed according to a sequence indicated by step numbers. Unlessotherwise explicitly specified in this application, execution of thesteps is not strictly limited, and the steps may be performed in othersequences. In addition, at least some of the steps in the foregoingembodiments may include a plurality of substeps or a plurality ofstages. These substeps or stages are not necessarily completed at thesame moment, but may be performed at different moments. Besides, thesesubsteps or stages may not be necessarily performed sequentially, butmay be performed in turn or alternately with other steps or at leastsome of substeps or stages of other steps.

To better implement the foregoing method, correspondingly, an embodimentof this application further provides an image segmentation apparatus.The image segmentation apparatus may be specifically integrated in anelectronic device. The electronic device may be a server or may be aterminal or may be a system including a terminal and a server.

For example, as shown in FIG. 3, the image segmentation apparatus mayinclude an obtaining unit 301, a first segmentation unit 302, a secondsegmentation unit 303, a determining unit 304, a training unit 305, anda third segmentation unit 306.

The obtaining unit 301 is configured to obtain a target domain image anda source domain image that is labeled with target information. In anembodiment, obtaining unit 301 is configured to obtain plural targetdomain images and plural source domain images that are labeled withtarget information.

The first segmentation unit 302 is configured to segment the sourcedomain image and the target domain image by using a generative networkin a first generative adversarial network to respectively determine afirst source domain segmentation loss and a first target domainsegmentation loss. In an embodiment, first segmentation unit 302 isconfigured to segment one or more of the source domain images and thetarget domain images by using a generative network in a first generativeadversarial network to respectively determine first source domainsegmentation losses and first target domain segmentation losses.

The second segmentation unit 303 is configured to segment the sourcedomain image and the target domain image by using a generative networkin a second generative adversarial network to respectively determine asecond source domain segmentation loss and a second target domainsegmentation loss. In an embodiment, second segmentation unit 303 isconfigured to segment one or more of the source domain images and thetarget domain images by using a generative network in a secondgenerative adversarial network to respectively determine second sourcedomain segmentation losses and second target domain segmentation losses.

The determining unit 304 is configured to: determine a first sourcedomain target image and a second source domain target image according tothe first source domain segmentation loss and the second source domainsegmentation loss, and determine a first target domain target image anda second target domain target image according to the first target domainsegmentation loss and the second target domain segmentation loss. In anembodiment, the determining unit 304 is configured to determine a firstsource domain image and a second source domain image according to thefirst source domain segmentation losses and the second source domainsegmentation losses, and determine a first target domain image and asecond target domain image according to the first target domainsegmentation losses and the second target domain segmentation losses.

The training unit 305 is configured to perform cross training on thefirst generative adversarial network and the second generativeadversarial network by using the first source domain target image, thefirst target domain target image, the second source domain target image,and the second target domain target image to obtain the first generativeadversarial network after training. In an embodiment, the training unit305 is configured to perform cross training on the first generativeadversarial network and the second generative adversarial network byusing the first source domain image, the first target domain image, thesecond source domain image, and the second target domain image to obtaina trained first generative adversarial network.

The third segmentation unit 306 is configured to segment ato-be-segmented image based on the generative network in the firstgenerative adversarial network after training to obtain a segmentationresult. In an embodiment, the third segmentation unit 306 is configuredto segment a to-be-segmented image based on the generative network inthe trained first generative adversarial network to obtain asegmentation result.

In some embodiments, the first segmentation unit 302 may include a firstextraction subunit, a first segmentation subunit, and a secondsegmentation subunit.

The first extraction subunit is configured to perform feature extractionon the source domain image and the target domain image by using thegenerative network in the first generative adversarial network torespectively obtain feature information of a first source domain imageand feature information of a first target domain image. In anembodiment, the first extraction subunit is configured to performfeature extraction on the source domain images and the target domainimages by using the generative network in the first generativeadversarial network to respectively obtain feature information of onesource domain image of the source domain images and feature informationof one target domain image of the target domain images.

The first segmentation subunit is configured to perform targetsegmentation on the source domain image based on the feature informationof the first source domain image to determine the first source domainsegmentation loss. In an embodiment, the first segmentation subunit isconfigured to perform segmentation on the one source domain image of thesource domain images based on the feature information of the one sourcedomain image to determine one of the first source domain segmentationlosses.

The second segmentation subunit is configured to perform targetsegmentation on the target domain image based on the feature informationof the first target domain image to determine the first target domainsegmentation loss. In an embodiment, the second segmentation subunit isconfigured to perform segmentation on the one target domain image of thetarget domain images based on the feature information of the one targetdomain image to determine one of the first target domain segmentationlosses.

In some embodiments, the source domain image includes a noisy image anda noiseless image, and the first segmentation subunit is specificallyconfigured to: perform segmentation on the noisy image in the sourcedomain image based on the feature information of the first source domainimage to obtain a first noise segmentation probability; obtain aweighted graph of the noisy image in the source domain image; obtain afirst noise segmentation loss according to the first noise segmentationprobability and the weighted graph of the noisy image; performsegmentation on the noiseless image in the source domain image based onthe feature information of the first source domain image to obtain afirst noiseless segmentation probability; obtain a first noiselesssegmentation loss according to the first noiseless segmentationprobability and a labeling result of the noiseless image; and determinethe first source domain segmentation loss based on the first noisesegmentation loss and the first noiseless segmentation loss.

In some embodiments, the second segmentation subunit is specificallyconfigured to: perform target segmentation on the target domain imagebased on the feature information of the first target domain image toobtain a first target domain segmentation probability; generate a firsttarget domain segmentation result according to the first target domainsegmentation probability; and obtain the first target domainsegmentation loss according to the first target domain segmentationresult and the target domain image.

In some embodiments, the second segmentation unit 303 may include asecond extraction subunit, a third segmentation subunit, and a fourthsegmentation subunit.

The second extraction subunit is configured to perform featureextraction on the source domain image and the target domain image byusing the generative network in the second generative adversarialnetwork to respectively obtain feature information of a second sourcedomain image and feature information of a second target domain image. Inan embodiment, the second extraction subunit is configured to performfeature extraction on one of the source domain images and on one of thetarget domain images by using the generative network in the secondgenerative adversarial network to respectively obtain featureinformation of the one of the second source domain images and featureinformation of the one of the second target domain images.

The third segmentation subunit is configured to perform targetsegmentation on the source domain image based on the feature informationof the second source domain image to determine the second source domainsegmentation loss. In an embodiment, the third segmentation subunit isconfigured to perform segmentation on the one of the source domainimages based on the feature information of the one of the source domainimages to determine one of the second source domain segmentation losses.

The fourth segmentation subunit is configured to perform targetsegmentation on the target domain image based on the feature informationof the second target domain image to determine the second target domainsegmentation loss. In an embodiment, the fourth segmentation subunit isconfigured to perform segmentation on the one of the target domainimages based on the feature information of the one of the target domainimages to determine one of the second target domain segmentation losses.

In some embodiments, the source domain image includes a noisy image anda noiseless image, and the third segmentation subunit is specificallyconfigured to: perform target segmentation on the noisy image in thesource domain image based on the feature information of the sourcedomain image to obtain a second noise segmentation probability; obtain aweighted graph of the noisy image in the source domain image; obtain asecond noise segmentation loss according to the second noisesegmentation probability and the weighted graph of the noisy image;perform target segmentation on the noiseless image in the source domainimage based on the feature information of the source domain image toobtain a second noiseless segmentation probability; obtain a secondnoiseless segmentation loss according to the second noiselesssegmentation probability and a labeling result of the noiseless image;and determine the second source domain segmentation loss based on thesecond noise segmentation loss and the second noiseless segmentationloss.

In some embodiments, the fourth segmentation subunit is specificallyconfigured to: perform target segmentation on the target domain imagebased on the feature information of the target domain image to obtain asecond target domain segmentation probability; generate a second targetdomain segmentation result according to the second target domainsegmentation probability; and obtain the second target domainsegmentation loss according to the second target domain segmentationresult and the target domain image.

In some embodiments, the determining unit 304 may include a firstdetermining subunit and a second determining subunit.

The first determining subunit may be specifically configured to: sortthe first source domain segmentation loss, select a source domain imagemeeting a preset loss condition according to the sorted first sourcedomain segmentation loss, and determine the source domain image as thefirst source domain target image; and sort the second source domainsegmentation loss, select a source domain image meeting the preset losscondition according to the sorted second source domain segmentationloss, and determine the source domain image as the second source domaintarget image.

The second determining subunit may be specifically configured to: trainthe first generative adversarial network according to the first targetdomain segmentation loss, and generate the first target domain targetimage by using a training result; and train the second generativeadversarial network according to the second target domain segmentationloss, and generate the second target domain target image by using atraining result.

In some embodiments, the training unit 305 may include a first trainingsubunit and a second training subunit.

The first training subunit may be specifically is configured to: segmentthe second source domain target image and the second target domaintarget image by using the generative network in the first generativeadversarial network to respectively obtain a second source domain targetsegmentation result and a second target domain target segmentationresult; discriminate the second source domain target segmentation resultand the second target domain target segmentation result by using adiscriminative network in the first generative adversarial network toobtain a second target discrimination result; and train the firstgenerative adversarial network according to the second source domaintarget segmentation result, the second target domain target segmentationresult, and the second target discrimination result to obtain the firstgenerative adversarial network after training.

The second training subunit may be specifically configured to: segmentthe first source domain target image and the first target domain targetimage by using the generative network in the second generativeadversarial network to respectively obtain a first source domain targetsegmentation result and a first target domain target segmentationresult; discriminate the first source domain target segmentation resultand the first target domain target segmentation result by using adiscriminative network in the second generative adversarial network toobtain a first target discrimination result; and train the secondgenerative adversarial network according to the first source domaintarget segmentation result, the first target domain target segmentationresult, and the first target discrimination result to obtain the secondgenerative adversarial network after training.

In some embodiments, the first training subunit may be specificallyconfigured to: calculate information entropy of the second target domaintarget image; and obtain the second target discrimination result byusing the discriminative network in the first generative adversarialnetwork and according to the second source domain target segmentationresult, the second target domain target segmentation result, and thesecond target domain target image.

In some embodiments, the first training subunit may be specificallyconfigured to: obtain a second source domain target segmentation lossaccording to the second source domain target segmentation result and alabeling result of the second source domain target image; obtain asecond target domain target segmentation loss according to the secondtarget domain target segmentation result and the second target domaintarget image; obtain a second target discrimination loss of thediscriminative network according to the second source domain targetsegmentation result and the second target domain target segmentationresult; and train the first generative adversarial network according tothe second source domain target segmentation loss, the second targetdomain target segmentation loss, and the second target discriminationloss to obtain the first generative adversarial network after training.

In some embodiments, the first training subunit may be specificallyconfigured to: build a minimal adversarial loss of the first generativeadversarial network according to the second source domain targetsegmentation loss and the second target domain target segmentation loss;build a maximal adversarial loss of the first generative adversarialnetwork according to the second target discrimination loss; and performiterative training on the first generative adversarial network based onthe minimal adversarial loss and the maximal adversarial loss to obtainthe first generative adversarial network after training.

In some embodiments, the second training subunit may be specificallyconfigured to: calculate information entropy of the first target domaintarget image; and obtain the first target discrimination result by usingthe discriminative network in the second generative adversarial networkand according to the first source domain target segmentation result, thefirst target domain target segmentation result, and the first targetdomain target image.

In some embodiments, the second training subunit may be specificallyconfigured to: obtain a first source domain target segmentation lossaccording to the first source domain target segmentation result and alabeling result of the first source domain target image; obtain a firsttarget domain target segmentation loss according to the first targetdomain target segmentation result and the first target domain targetimage; obtain a first target discrimination loss of the discriminativenetwork according to the first source domain target segmentation resultand the first target domain target segmentation result; and train thesecond generative adversarial network according to the first sourcedomain target segmentation loss, the first target domain targetsegmentation loss, and the first target discrimination loss to obtainthe second generative adversarial network after training.

In some embodiments, the second training subunit may be specificallyconfigured to: build a minimal adversarial loss of the second generativeadversarial network according to the first source domain targetsegmentation loss and the first target domain target segmentation loss;build a maximal adversarial loss of the second generative adversarialnetwork according to the first target discrimination loss; and performiterative training on the second generative adversarial network based onthe minimal adversarial loss and the maximal adversarial loss to obtainthe second generative adversarial network after training.

During specific implementations, the foregoing units may be implementedas independent entities, or may be combined, or may be implemented asthe same entity or several entities. For specific implementations of theforegoing units, refer to the foregoing method embodiments. Details arenot described herein again.

As can be seen from above, in embodiments of this application, theobtaining unit 301 first obtains target domain images and source domainimages that are labeled with target information, the first segmentationunit 302 then segments the source domain images and the target domainimages by using a generative network in a first generative adversarialnetwork to respectively determine first source domain segmentationlosses and first target domain segmentation losses, the secondsegmentation unit 303 segments the source domain image and the targetdomain image by using a generative network in a second generativeadversarial network to respectively determine second source domainsegmentation losses and second target domain segmentation losses, next,the determining unit 304 determines a first source domain image and asecond source domain image according to the first source domainsegmentation losses and the second source domain segmentation losses,and determines a first target domain image and a second target domainimage according to the first target domain segmentation losses and thesecond target domain segmentation losses, the training unit 305 thenperforms cross training on the first generative adversarial network andthe second generative adversarial network by using the first sourcedomain image, the first target domain image, the second source domainimage, and the second target domain image to obtain the first generativeadversarial network after training, and then the third segmentation unit306 segments a to-be-segmented image based on the generative network inthe first generative adversarial network after training to obtain asegmentation result. The solution provides an unsupervised robustsegmentation method based on a domain adaptive strategy for thephenomenon that there is noise in labels of data and a distributiondifference between source domain and target domain datasets. Therefore,in the manner of mutual learning and mutual supervision between twomodels, labels with noise and an unsupervised image segmentation taskare resolved, thereby effectively improving the accuracy of imagesegmentation.

In addition, an embodiment of this application further provides anelectronic device. FIG. 4 is a schematic structural diagram of anelectronic device according to an embodiment of this application.

The electronic device may include components such as a processor 401(processing circuitry) with one or more processing cores, a memory 402(non-transitory computer-readable storage medium) with one or morecomputer-readable storage media, a power supply 403, and an input unit404. A person skilled in the art may understand that the electronicdevice structure shown in FIG. 4 does not constitute a limitation to theelectronic device. The electronic device may include more or fewercomponents than those shown in the figure, or some components may becombined, or a different component deployment may be used.

The processor 401 is a control center of the electronic device, andconnects various parts of the entire electronic device by using variousinterfaces and lines. By running or executing a software program and/ora module stored in the memory 402, and invoking data stored in thememory 402, the processor performs various functions of the electronicdevice and processes data, thereby performing overall monitoring on theelectronic device.

The processor 401 may include one or more processing cores. Preferably,the processor 401 may integrate an application processor and acommunication interface, where the application processor mainlyprocesses an operating system, a user interface, and an applicationprogram and the like, and the communication interface mainly processeswireless communication. It may be understood that the foregoingcommunication interface may not be integrated into the processor 401.

The memory 402 may be configured to store the software programs andmodules. The processor 401 runs the software programs and modules storedin the memory 402, to perform various function application and dataprocessing, The memory 402 may mainly include a program storage area anda data storage area. The program storage area may store the operatingsystem, an application program required by at least one function (suchas a sound playback function and an image display function), and thelike. The data storage area may store data created according to use ofthe electronic device, and the like. In addition, the memory 402 mayinclude a high speed random access memory, and may further include anon-volatile memory, such as at least one magnetic disk storage device,a flash memory or another volatile solid-state storage device.Correspondingly, the memory 402 may further include a memory controller,to provide access of the processor 401 to the memory 402.

The electronic device further includes the power supply 403 forsupplying power to the components. Preferably, the power supply 403 mayconnect to the processor 401 by using a power supply management system,thereby implementing functions, such as charging, discharging, and powerconsumption management, by using the power supply management system. Thepower supply 403 may further include one or more of a direct current oralternating current power supply, a recharging system, a power failuredetection circuit, a power supply converter or inverter, a power supplystate indicator, and any other component.

The electronic device may further include the input unit 404. The inputunit 404 may be configured to receive inputted numeric or characterinformation and generate keyboard, mouse, joystick, optical, ortrackball signal input related to user settings and function control.

Although not shown in the figure, the electronic device may furtherinclude a display unit, and the like.

Specifically, in this embodiment, the memory 402 in the electronicdevice stores computer-readable instructions capable of being run on theprocessor 401, and the processor 401 implements the following steps whenexecuting the computer-readable instructions: obtaining target domainimages and source domain images that are labeled with targetinformation, then segmenting the source domain images and the targetdomain images by using a generative network in a first generativeadversarial network to respectively determine first source domainsegmentation losses and first target domain segmentation losses,segmenting the source domain images and the target domain images byusing a generative network in a second generative adversarial network torespectively determine second source domain segmentation losses andsecond target domain segmentation losses, next, determining a firstsource domain image and a second source domain image according to thefirst source domain segmentation losses and the second source domainsegmentation losses, determining a first target domain image and asecond target domain image according to the first target domainsegmentation losses and the second target domain segmentation losses,then performing cross training on the first generative adversarialnetwork and the second generative adversarial network by using the firstsource domain image, the first target domain image, the second sourcedomain image, and the second target domain image to obtain the firstgenerative adversarial network after training, and then segmenting ato-be-segmented image based on the generative network in the firstgenerative adversarial network after training to obtain a segmentationresult.

For specific implementations of the above operations, refer to theforegoing embodiments.

As can be seen from above, in embodiments of this application, targetdomain images and source domain images that are labeled with targetinformation are obtained first, then the source domain images and thetarget domain images are segmented by using a generative network in afirst generative adversarial network to respectively determine firstsource domain segmentation losses and first target domain segmentationlosses, the source domain images and the target domain images aresegmented by using a generative network in a second generativeadversarial network to respectively determine second source domainsegmentation losses and second target domain segmentation losses, next,a first source domain image and a second source domain image aredetermined according to the first source domain segmentation losses andthe second source domain segmentation losses, a first target domainimage and a second target domain image are determined according to thefirst target domain segmentation losses and the second target domainsegmentation losses, then cross training is performed on the firstgenerative adversarial network and the second generative adversarialnetwork by using the first source domain image, the first target domainimage, the second source domain image, and the second target domainimage to obtain the first generative adversarial network after training,and a to-be-segmented image is then segmented based on the generativenetwork in the first generative adversarial network after training toobtain a segmentation result. The solution provides an unsupervisedrobust segmentation method based on a domain adaptive strategy for thephenomenon that there is noise in labels of data and a distributiondifference between source domain and target domain datasets. Therefore,in the manner of mutual learning and mutual supervision between twomodels, labels with noise and an unsupervised image segmentation taskare resolved, thereby effectively improving the accuracy of imagesegmentation.

A person skilled in the art may understand that all or part of the stepsin the various methods of the foregoing embodiments may be completed byusing the computer readable instruction or completed by using thecomputer readable instruction to control related hardware. The computerreadable instruction may be stored in a non-volatile storage medium(non-transitory computer-readable storage medium), loaded and executedby the processor.

In view of this, an embodiment of this application further provides oneor more non-volatile storage media storing a computer readableinstruction, the computer readable instruction, when executed by one ormore processors, causing the processor to perform the following steps:obtaining target domain images and source domain images that are labeledwith target information, then segmenting the source domain images andthe target domain images by using a generative network in a firstgenerative adversarial network to respectively determine first sourcedomain segmentation losses and first target domain segmentation losses,segmenting the source domain images and the target domain images byusing a generative network in a second generative adversarial network torespectively determine second source domain segmentation losses andsecond target domain segmentation losses, next, determining a firstsource domain image and a second source domain image according to thefirst source domain segmentation losses and the second source domainsegmentation losses, determining a first target domain image and asecond target domain image according to the first target domainsegmentation losses and the second target domain segmentation losses,then performing cross training on the first generative adversarialnetwork and the second generative adversarial network by using the firstsource domain image, the first target domain image, the second sourcedomain image, and the second target domain image to obtain the firstgenerative adversarial network after training, and then segmenting ato-be-segmented image based on the generative network in the firstgenerative adversarial network after training to obtain a segmentationresult.

For specific implementations of the above operations, refer to theforegoing embodiments.

The non-volatile storage medium may include a read-only memory (ROM), arandom access memory (RAM), a magnetic disk, an optical disc or thelike.

The image segmentation method and apparatus and the storage mediumprovided in the embodiments of this application are described above indetail. Although the principles and implementations of this applicationare described by using specific examples in this specification, thedescriptions of the foregoing embodiments are merely intended to helpunderstand the method and the core idea of the method of thisapplication. Meanwhile, a person skilled in the art may makemodifications to the specific implementations and application rangeaccording to the idea of this application. In conclusion, the content ofthis specification is not to be construed as a limitation to thisapplication.

What is claimed is:
 1. An image segmentation method, comprising:obtaining plural target domain images and plural source domain imagesthat are labeled with target information: segmenting one or more of thesource domain images and the target domain images by using a generativenetwork in a first generative adversarial network to respectivelydetermine first source domain segmentation losses and first targetdomain segmentation losses: segmenting one or more of the, source domainimages and the target domain images by using a generative network in asecond generative adversarial network to respectively determine secondsource domain segmentation losses and second target domain segmentationlosses; determining a first source domain image and a second sourcedomain image according to the first source domain segmentation lossesand the second source domain segmentation losses, and determining a.first target domain image and a second target domain image according tothe first target domain segmentation losses and the second target &mainsegmentation losses: performing cross training on the first generativeadversarial network and the second generative adversarial network byusing the first source domain image, the first target domain image, thesecond source domain image, and the second target domain image to obtaina trained first generative adversarial network; and segmenting ato-be-segmented image based on the generative network in the trainedfirst generative adversarial network to obtain a segmentation result. 2.The method according to claim 1, wherein the segmenting die sourcedomain images and the target domain images comprises: performing featureextraction on the source domain images and the target domain images byusing the generative network in the first generative adversarial networkto respectively obtain feature information of one source domain image ofthe source domain images and feature information of one target domainimage of the target domain images; peforming segmentation on the onesource domain image of the source domain images based on the featureinformation of the one source domain image to determine one of the firstsource domain segmentation losses; and performing segmentation on theone target domain image of the target domain images based on the featureinformation of the one target domain image to determine one of the firsttarget domain segmentation losses.
 3. The method according to claim 2,wherein each of the source domain images comprises a noisy image and anoiseless image, and the performing the segmentation on the one sourcedomain image based on the feature information of the one source domainimage to determine one of the first source domain segmentation lossescomprises: performing segmentation on the noisy image in the one sourcedome in image based on the feat information of the one source domainimage to obtain a first noise segmentation probability obtaining aweighted graph of the image in the one source domain image; obtaining afirst noise segmentation loss according to the first noise segmentationprobability and the weighted graph of the noisy image; performingsegmentation on the noiseless image in the one source domain image basedon the feature information of the one source domain image to obtain afirst noiseless segmentation probability; obtaining a first noiselesssegmentation loss according to the first noiseless segmentationprobability and a labeling result of the noiseless image; anddetermining the one of the first source domain segmentation losses basedon the first noise segmentation loss and the first noiselesssegmentation loss.
 4. The method according to claim 2, wherein theperforming the segmentation on the one target domain image based on thefeature information of the one target domain image to determine one ofthe first target domain segmentation losses comprises: performingsegmentation on the one target domain image based on the featureinformation of the one target domain image to obtain a first targetdomain segmentation probability; generating a first target domainsegmentation result according to the first target domain segmentationprobability; and obtaining the one of the first target domainsegmentation losses according to the first target domain segmentationresult and the one target domain image.
 5. The method according to claim1, wherein the segmenting the source domain images and the target domainimages by using a generative network in a second generative adversarialnetwork to respectively determine second source domain segmentationlosses and second target domain segmentation losses comprises;performing, feature extraction on one of the source domain images and onone of the target domain images by using the generative network in thesecond generative adversarial network to respectively obtain featureinformation of the one of the source domain images and featureinformation of the one of the target domain images; performingsegmentation on the one of the source domain images based on the featureinformation of the one of the source domain images to determine one ofthe second source domain segmentation losses; and performingsegmentation on the one of the target domain images based on the featureinformation of the one of the target domain images to determine one ofthe second target domain segmentation losses.
 6. The method according toclaim 1, wherein the determining comprises: sorting the first sourcedomain segmentation losses, selecting a source domain image from amongthe source domain images meeting a preset loss condition according tothe sorted first source domain segmentation losses, and determining theselected source domain image as the first source domain image; andsorting the second source domain segmentation losses, selecting a sourcedomain image from among the source domain images meeting the preset losscondition according to the sorted second source domain segmentationlosses, and determining the selected source domain image as the secondsource domain image.
 7. The method according to claim 1, wherein thedetermining comprises: training the first generative adversarial networkaccording to the first target domain segmentation losses, and generatingthe first target domain image by using a training result; and trainingthe second generative adversarial network according to the second targetdomain segmentation losses, and generating the second target domainimage by using a training result.
 8. The method according to claim 1,wherein the performing the cross training comprises: segmenting thesecond source domain image and the second target domain image by usingthe generative network in the first generative adversarial network torespectively obtain a second source domain segmentation result and asecond target domain segmentation result; discriminating the secondsource domain segmentation result and the second target domainsegmentation result by using a discriminative network in the firstgenerative adversarial network to obtain a second discrimination result;training the first generative adversarial network according to thesecond source domain segmentation result, the second target domainsegmentation result, and the second discrimination result to obtain thetrained. first generative adversarial network: segmenting the firstsource domain image and the first target domain image by using thegenerative network in the second generative adversarial network torespectively obtain a first source domain segmentation result and afirst target domain segmentation result, discriminating the first sourcedomain segmentation result and the first target domain segmentationresult by using a discriminative network in the second generativeadversarial network to obtain a first discrimination result, andtraining the second generative adversarial network according to thefirst source domain segmentation result, the first target domainsegmentation result, and the first discrimination result to obtain atrained second generative adversarial network.
 9. The method accordingto claim 8, wherein the discriminating the second source domainsegmentation result and the second target domain segmentation result byusing the discriminative network in the first generative adversarialnetwork to obtain the second discrimination result comprises:calculating information entropy of the second target domain image; andobtaining the second discrimination result by using the discriminativenetwork in the first generative adversarial network and according to thesecond source domain segmentation result, the second target domainsegmentation result, and the second target domain image.
 10. The methodaccording to claim 8, wherein the training the first generativeadversarial network comprises: obtaining one of the second source domainsegmentation losses according to the second source domain segmentationresult and a labeling result of the second source domain image;obtaining one of the second target domain segmentation losses accordingto the second target domain segmentation result and the second targetdomain image; obtaining a second discrimination loss of thediscriminative network according to the second source domainsegmentation result and the second target domain segmentation result;and training the first generative adversarial network according to theone of the second source domain segmentation losses, the one of thesecond target domain segmentation losses, and the second discriminationloss to obtain the trained first generative adversarial network.
 11. Themethod according to claim 10, wherein the training the first generativeadversarial network according to the one of the second source domainsegmentation losses, the one of the second target domain segmentationlosses, and the second discrimination loss to obtain the trained firstgenerative adversarial network comprises; building a minimal adversarialloss of the first generative adversarial network according to the one ofthe second source domain segmentation losses and the one or the secondtarget domain segmentation losses; building a maximal adversarial lossof the first generative adversarial network according to the seconddiscrimination loss; and performing iterative training on the firstgenerative adversarial network based on the minimal adversarial loss andthe maximal adversarial loss to obtain the trained first generativeadversarial network.
 12. An image segmentation apparatus, comprising:processing circuitry configured to obtain plural target domain imagesand plural source domain images that are labeled with targetinformation; segment one or more of the source domain images and thetarget. domain Images by using a generative network in a firstgenerative adversarial network to respectively determine first sourcedomain segmentation losses and first target domain segmentation losses;segment one or more of the source domain images and the target domainimages by using a generative network in a second generative adversarialnetwork to respectively determine second source domain segmentationlosses and second target domain segmentation losses; determine a firstsource domain image and a second source domain image according to thefirst source domain segmentation losses and the second source domainsegmentation losses, and determine a first target domain image and asecond target domain image according to the first target domainsegmentation losses and the second target domain segmentation losses;perform cross training on the first generative adversarial network andthe second generative adversarial network by using the first sourcedomain image, the first target domain image, the second source domainimage, and the second target domain image to obtain a trained firstgenerative adversarial network; and segment a to-be-segmented imagebased on the generative network in the trained first generativeadversarial network to obtain a segmentation result.
 13. The imagesegmentation apparatus according to claim wherein the processingcircuitry is configured to segment the source domain images and thetarget main images by performing feature extraction ori the sourcedomain images and the target domain images by using the generativenetwork in the first generative adversarial network to respectivelyobtain feature information of one source domain image of the sourcedomain images and feature information of one target domain image of thetarget domain images; performing segmentation on the one source domainimage of the source domain images based on the feature Information ofthe one source domain to determine one of the first source domainsegmentation losses; and performing segmentation on the target domainimage of the target domain images based on the feature information ofthe one target domain image to determine one of the first target domainsegmentation losses.
 14. The image segmentation apparatus according toclaim 13, wherein each of the source domain images comprises a noisyimage and a noiseless image, and the processing circuitry is configuredto perform the, segmentation on the one source domain image based on thefeature illumination of the one source domain image to determine one ofthe first source domain segmentation losses by performing segmentationon the noisy image in the one source domain image based on the featureinformation of the one source domain image to obtain a first noisesegmentation probability; obtaining a weighted graph of the noisy imagein the one source domain image; obtaining a first noise segmentationloss according to the first noise segmentation probability and theweighted graph of the noisy image; performing segmentation on thenoiseless image in the one source domain image based on the featureinformation of the one source domain image to obtain a first noiselesssegmentation probability; obtaining a first noiseless segmentation lossaccording to the first noiseless segmentation probability and a labelingresult of the noiseless image; and determining the one of the firstsource domain segmentation losses based on the first noise segmentationloss and the first noiseless segmentation loss.
 15. The imagesegmentation apparatus according to claim 13, wherein the processingcircuitry is configured to perform the segmentation on the one targetdomain image based on the feature information of the one target domainimage to determine one of the first target domain segmentation losses byperforming segmentation on the one target domain image based on thefeature information of the one target domain image to obtain a firsttarget domain segmentation probability; generating a first target domainsegmentation result according to the first target domain segmentationprobability; and obtaining the one of the first target domainsegmentation losses according to the first target domain segmentationresult and the one target domain image.
 16. The image segmentationapparatus according to claim 12, wherein the processing circuitry isconfigured to segment the source domain images and the target domainimages by using a generative network in a second generative adversarialnetwork to respectively determine second source domain segmentationlosses and second. target domain segmentation losses by performingfeature extraction on one of the source domain images and on one of thetarget domain images by using the generative network in the secondgenerative adversarial network to respectively obtain featureinformation of the one of the source domain images and featureinformation of the one of die target domain images; performingsegmentation on the one of the source domain images based on the featureinformation or the one of the source domain images to determine one ofthe second source domain segmentation losses; and performingsegmentation on the one of the target domain images based on the featureinformation of the one of the target domain images to determine one ofthe second target domain segmentation losses.
 17. The image segmentationapparatus according to claim 12, wherein the processing circuitry isconfigured to perform the determining by sorting the first source domainsegmentation losses, selecting a source domain image from among thesource domain images meeting a preset loss condition according to thesorted first source domain segmentation losses, and determining theselected source domain image as the first source domain image; andsorting the second source domain segmentation losses, selecting a sourcedomain image from among the source domain images meeting the preset losscondition according to the sorted second source domain segmentationlosses, and determining the selected source domain image as the secondsource domain image.
 18. The image segmentation apparatus according toclaim 12 wherein the processing circuitry is configured to perform thedetermining by training the first generative adversarial networkaccording to the first target domain segmentation losses, and generatingthe first target domain image by using a training result; and trainingthe second generative adversarial network according to the second targetdomain segmentation losses, and generating the second target domainimage by using a training result,
 19. The image segmentation apparatusaccording to claim 2, wherein the processing circuitry is configured toperform the cross training by segmenting the second source domain imageand the second target domain image by using the generative network inthe first generative adversarial network to respectively obtain a secondsource domain segmentation result and a second target domainsegmentation result; discriminating the second source domainsegmentation result and the second target domain segmentation result byusing a discriminative network in the first generative adversarialnetwork to obtain a second discrimination result; training the firstgenerative adversarial network according to the second source domainsegmentation result, the second target domain segmentation result, andthe second discrimination result to obtain the trained first generativeadversarial network; segmenting the first source domain image and thefirst target domain image by using the generative network in the secondgenerative adversarial network to respectively obtain a first sourcedomain segmentation result and a first target domain segmentationresult; discriminating the first source domain segmentation result andthe first target domain segmentation result by using a discriminativenetwork in the second generative adversarial network to obtain a firstdiscrimination result; and training the second generative adversarialnetwork according to the first source domain segmentation result, thefirst target domain segmentation result, and the first discriminationresult to obtain a trained second generative adversarial network.
 20. Anon-transitory computer-readable storage medium storingcomputer-readable instructions thereon which, when executed by aprocessor, cause the processor to perform an image segmentation methodcomprising: obtaining plural target domain images and plural sourcedomain images that are labeled with target information; segmenting oneor more of the source domain images and the target domain images byusing a generative network in a first generative adversarial network torespectively determine first source domain segmentation losses and firsttarget domain segmentation losses; segmenting one or more of the sourcedomain images and the target domain images by using a generative networkin a second generative adversarial network to respectively determinesecond source domain segmentation losses and second target domainsegmentation losses; determining a first source domain image and asecond source domain image according to the first source domainsegmentation losses and the second source domain segmentation losses,and determining a first target domain image and a second target domainimage according to the first target domain segmentation losses and thesecond target domain segmentation losses; performing cross training onthe first generative adversarial network and the second generativeadversarial network by using the first source domain image, the firsttarget domain image, the second source domain image, and the secondtarget domain image to obtain a trained first generative adversarialnetwork; and segmenting a to-be-segmented image based on the generativenetwork in the trained first generative adversarial network to obtain asegmentation result.